Image display system, non-transitory storage medium having stored therein image display program, display control apparatus, and image display method

ABSTRACT

An example of an image display system includes a goggle apparatus having a display section. A virtual camera and a user interface are placed in a virtual space. The orientation of the virtual camera in the virtual space is controlled in accordance with the orientation of the goggle apparatus. When the goggle apparatus rotates by an angle greater than or equal to a predetermined angle in a pitch direction, the user interface is moved to the front of the virtual camera in a yaw direction.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/808,742, filed on Mar. 4, 2020, and claims priority to JapanesePatent Application No. 2019-053214, filed on Mar. 20, 2019. Thedisclosure of each of these applications is incorporated herein byreference in their entirety.

FIELD

Exemplary embodiments relate to an image display system, anon-transitory storage medium having stored therein an image displayprogram, a display control apparatus, and an image display method thatare capable of displaying an image.

BACKGROUND AND SUMMARY

As related art, there is an image display system that places an objectin a virtual space and directs the line of sight of a virtual camera tothe object while moving in the virtual space, thereby displaying theobject.

In the related art, however, to display the object, for example, a userneeds to perform the operation of directing the line of sight of thevirtual camera to the object. There is room for improvement in improvingthe operability of the object.

Therefore, it is an object of an exemplary embodiment to provide animage display system, an image display program, a display controlapparatus, and an image display method that are capable of improving theoperability of an object placed in a virtual space.

To achieve the above object, the exemplary embodiment employs thefollowing configurations.

An image display system according to the exemplary embodiment includes agoggle apparatus and at least one processor. The at least one processorconfigured to at least: place an object in a virtual space; display, ona display section of the goggle apparatus, an image captured by avirtual camera in the virtual space; acquire at least a rotational anglein a yaw direction and a rotational angle in a pitch direction of thegoggle apparatus; rotate the virtual camera in a yaw direction in thevirtual space in accordance with the rotational angle in the yawdirection of the goggle apparatus and rotate the virtual camera in apitch direction in the virtual space in accordance with the rotationalangle in the pitch direction of the goggle apparatus; and if at leastthe rotational angle in the pitch direction of the goggle apparatus isoutside a range, perform at least either one of a movement of the objectand control of the virtual camera in the virtual space so that theobject is located in front of the virtual camera in the yaw direction.

Based on the above, when a rotational angle in a pitch direction of agoggle apparatus is outside a range, for example, it is possible to movean object or control a virtual camera so that the object is located infront of the virtual camera in a yaw direction. Consequently, forexample, when the goggle apparatus is rotated by an angle greater thanor equal to a predetermined angle in the pitch direction, a user caneasily view the object from the front. Thus, it is possible to improvethe operability of the object.

In another configuration, the at least one processor may be configuredto at least, while the rotational angle in the pitch direction of thegoggle apparatus is outside the range, perform at least either one ofthe movement of the object and the control of the virtual camera inaccordance with rotation of the goggle apparatus in the yaw direction sothat the object continues to be located in front of the virtual camerain the yaw direction.

Based on the above, for example, even when the goggle apparatus isrotated in the yaw direction while the rotational angle in the pitchdirection of the goggle apparatus is outside the range, it is possibleto locate the object in front in the yaw direction.

In another configuration, the at least one processor may be configuredto at least, if the rotational angle in the pitch direction of thegoggle apparatus is in the range, not perform the movement of the objectand the control of the virtual camera to locate the object in front ofthe virtual camera in the yaw direction.

Based on the above, if the rotational angle in the pitch direction ofthe goggle apparatus is in the range, for example, it is possible tocontinue to maintain the position of the object.

In another configuration, the range may be a range including such anangle that the goggle apparatus is in a horizontal state.

Based on the above, for example, when the goggle apparatus is in ahorizontal state, for example, it is possible to continue to maintainthe position of the object.

In another configuration, the at least one processor may be configuredto at least: select the object at least based on the rotational angle inthe pitch direction of the goggle apparatus; and if the rotational anglein the pitch direction of the goggle apparatus is outside the range, notselect the object.

Based on the above, it is possible to select the object in accordancewith the rotational angle of the goggle apparatus. Thus, in the statewhere the object cannot be selected, it is possible to perform at leasteither one of a movement of the object and control of the virtualcamera.

In another configuration, the at least one processor may be configuredto at least: determine whether or not the rotational angle in the yawdirection of the goggle apparatus satisfies a predetermined condition;and if it is determined that the rotational angle in the yaw directionof the goggle apparatus satisfies the predetermined condition, performat least either one of the movement of the object and the control of thevirtual camera in the virtual space so that the object is located infront of the virtual camera in the yaw direction.

Based on the above, further based on a rotational angle in a yawdirection of the goggle apparatus, it is possible to perform a movementof an object or control of the virtual camera.

In another configuration, the at least one processor may be configuredto at least, if an angle of depression of the goggle apparatus isgreater than or equal to a threshold, or when an angle of elevation ofthe goggle apparatus is greater than or equal to a threshold, determinethat the rotational angle in the pitch direction of the goggle apparatusis outside the range.

Based on the above, when the goggle apparatus is directed downward orupward, it is possible to perform a movement of an object or control ofthe virtual camera. If an angle of depression of the goggle apparatus isa threshold, it may be determined that the rotational angle in the pitchdirection of the goggle apparatus is outside the range, or it may bedetermined that the rotational angle in the pitch direction of thegoggle apparatus is within the range. When an angle of elevation of thegoggle apparatus is a threshold, it may be determined that therotational angle in the pitch direction of the goggle apparatus isoutside the range, or it may be determined that the rotational angle inthe pitch direction of the goggle apparatus is within the range.

In another configuration, the at least one processor may be configuredto at least, if the rotational angle in the pitch direction of thegoggle apparatus is outside the range, move the object to the front ofthe virtual camera in the yaw direction while maintaining a position ofthe object in a height direction in the virtual space.

Based on the above, it is possible to move the object to the front ofthe virtual camera in the yaw direction while maintaining the positionof the object in a height direction. Consequently, when a user faces thefront, the user can view the object from the front. Thus, it is possibleto improve the operability of the object.

In another configuration, the object may be a user interface that can beoperated by a user.

Based on the above, it is possible to improve the operability of a userinterface.

In another configuration, the at least one processor may be configuredto at least, in a case where the rotational angle in the pitch directionof the goggle apparatus is outside the range, and then further, when therotational angle in the pitch direction of the goggle apparatus changesfrom outside the range to within the range, perform at least either oneof the movement of the object and the control of the virtual camera sothat the object is in front of the virtual camera in the yaw direction.

Based on the above, if the rotational angle in the pitch direction ofthe goggle apparatus is outside the range and is then within the range,it is possible to locate the object to the front of the virtual camerain the yaw direction. Consequently, for example, when a user rotates thegoggle apparatus by an angle greater than or equal to a predeterminedangle in the pitch direction, then returns the rotation in the pitchdirection of the goggle apparatus, and views the goggle apparatus, theuser can view the object from the front.

Another exemplary embodiment may be an image display program that causesa processor of an apparatus that displays an image on a display sectionof a goggle apparatus to execute the above processing. Another exemplaryembodiment may be a display control apparatus that displays an image ona display section of a goggle apparatus, and also may be an apparatusthat executes the above processing. Another exemplary embodiment may bean image display method performed by the image display system includinga goggle apparatus.

According to the exemplary embodiment, for example, by rotating a goggleapparatus in a pitch direction, it is possible to move an object to thefront of a virtual camera in a yaw direction. Thus, it is possible toimprove the operability of the object.

These and other objects, features, aspects and advantages of theexemplary embodiments will become more apparent from the followingdetailed description of the exemplary embodiments when taken inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing an example non-limiting image display system1 according to the exemplary embodiment;

FIG. 2 is a diagram showing an example non-limiting external appearanceof an information processing apparatus 2 included in the image displaysystem 1 and including a display section 21;

FIG. 3 is a diagram showing an example non-limiting functionalconfiguration of the information processing apparatus 2;

FIG. 4 is a diagram showing an example non-limiting state where a useruses a goggle apparatus 10;

FIG. 5 is a diagram showing an example non-limiting virtual space VS;

FIG. 6A is a diagram showing an example non-limiting virtual spaceviewed from above when the goggle apparatus 10 is maintained in areference orientation;

FIG. 6B is a diagram showing an example non-limiting virtual spaceviewed from above when the goggle apparatus 10 is rotated in a yawdirection (a left direction) from the reference orientation;

FIG. 6C is a diagram showing an example non-limiting virtual spaceviewed from above when the goggle apparatus 10 is further rotated in theyaw direction (the left direction) from the state in FIG. 6B;

FIG. 7A is a diagram showing an example non-limiting image viewed by theuser, and also an example non-limiting image of the virtual space viewedfrom a virtual camera VC (VCL or VCR) in the orientation shown in FIG.6A;

FIG. 7B is a diagram showing an example non-limiting image viewed by theuser, and also an example non-limiting image of the virtual space viewedfrom the virtual camera VC in the orientation shown in FIG. 6B;

FIG. 7C is a diagram showing an example non-limiting image viewed by theuser, and also an example non-limiting image of the virtual space viewedfrom a virtual camera VC in the orientation shown in FIG. 6C;

FIG. 8 is a diagram showing an example non-limiting state where a UIobject 30 moves to the front of the virtual camera VC when the goggleapparatus 10 is reversed by a predetermined angle in the right directionin the orientation shown in FIG. 6C;

FIG. 9 is a diagram showing example non-limiting areas to which the UIobject 30 is moved;

FIG. 10 is a diagram showing an example non-limiting virtual camera VCviewed from above the virtual space and is also a diagram showing anexample non-limiting state where the virtual camera VC is rotated in theleft direction and then rotated in the right direction;

FIG. 11 is a diagram showing an example non-limiting virtual camera VCviewed from the horizontal direction in the virtual space and is also adiagram showing an example non-limiting state where the virtual cameraVC is rotated in the down direction and then rotated in the updirection;

FIG. 12 is a diagram showing an example non-limiting state where the UIobject 30 moves in a case where the line of sight of the virtual cameraVC is present in an A-area;

FIG. 13A is a diagram showing an example non-limiting image displayed onthe display section 21 in a case where the line of sight of the virtualcamera VC rotates in the down direction and enters the A-area;

FIG. 13B is a diagram showing an example non-limiting image displayed onthe display section 21 when the virtual camera VC rotates in the leftdirection from the state in FIG. 13A;

FIG. 14 is a diagram showing example non-limiting data stored in (a DRAM23 of) the information processing apparatus 2;

FIG. 15 is a flow chart showing an example non-limiting main processperformed by a processor 20 of the information processing apparatus 2;and

FIG. 16 is a flow chart showing an example non-limiting movement processin a B-area in step S106.

DETAILED DESCRIPTION OF NON-LIMITING EXAMPLE EMBODIMENTS

(Configuration of Image Display System) With reference to the drawings,an image display system 1 according to an exemplary embodiment isdescribed below. The image display system 1 according to the exemplaryembodiment allows a user to experience virtual reality (VR). FIG. 1 is adiagram showing an example of the image display system 1 according tothe exemplary embodiment. FIG. 2 is a diagram showing an example of theexternal appearance of an information processing apparatus 2 included inthe image display system 1 and including a display section 21.

As shown in FIG. 1, the image display system 1 includes a goggleapparatus 10. The goggle apparatus 10 is held by both hands or one handof the user and put on the face of the user by covering the left andright eyes of the user.

The goggle apparatus 10 includes an upper surface 11, a right sidesurface 12, a lower surface 13, a left side surface 14, and a bottomsurface 15. The surfaces 11 to 15 form a goggle main body. Further, onthe bottom surface 15 of the goggle apparatus 10, an approximatelycircular left opening 16L is provided at a position corresponding to theleft eye of the user, and an approximately circular right opening 16R isprovided at a position corresponding to the right eye of the user. Alens is fitted to each of the left opening 16L and the right opening16R. Further, in an inner space surrounded by the surfaces 11 to 15 ofthe goggle apparatus 10, a partition surface 17 is provided thatseparates the inner space of the goggle apparatus 10 into left and rightportions.

The goggle apparatus 10 includes a display section 21. Specifically, onthe back side of the bottom surface 15 of the goggle main body, theinformation processing apparatus 2 including the display section 21 isprovided. A part of the display section 21 of the information processingapparatus 2 provided on the back side of the bottom surface 15 is viewedby the user from the left opening 16L and the right opening 16R throughlenses. As shown in FIG. 2, the display section 21 is a horizontallylong and approximately rectangular display screen. A left area 21Lsurrounded by a dashed line on the left side of FIG. 2 is viewed by theleft eye of the user through the lens provided in the left opening 16L,and a right area 21R surrounded by a dashed line on the right side isviewed by the right eye of the user through the lens provided in theright opening 16R. When the user holds the goggle apparatus 10 and putsthe goggle apparatus 10 on their face, the left eye of the user isalmost surrounded by the upper surface 11, the lower surface 13, theleft side surface 14, and the partition surface 17, and the right eye ofthe user is almost surrounded by the upper surface 11, the lower surface13, the right side surface 12, and the partition surface 17. Thus, aleft eye image displayed in the left area 21L of the display section 21is viewed by the left eye of the user, while the surrounding environmentand a right eye image displayed in the right area 21R are difficult forthe left eye to view. The right eye image displayed in the right area21R of the display section 21 is viewed by the right eye of the user,while the surrounding environment and the left eye image displayed inthe left area 21L are difficult for the right eye to view.

The goggle main body and the display section 21 form the goggleapparatus 10. The information processing apparatus 2 including thedisplay section 21 is detachably attached to the goggle main body. Asthe information processing apparatus 2 attachable to and detachable fromthe goggle main body, a tablet terminal, a smartphone, a mobile gameapparatus, or the like may be used.

Although the details will be described below, the information processingapparatus 2 generates a left eye image obtained by viewing a virtualspace from a left virtual camera, and a right eye image obtained byviewing the virtual space from the right virtual camera. The informationprocessing apparatus 2 displays the generated left eye image and righteye image in the left area 21L and the right area 21R, respectively, ofthe display section 21. This enables the user to view a stereoscopicimage and experience virtual reality (VR) as if the user themselves waspresent in the virtual space.

FIG. 3 is a diagram showing an example of the functional configurationof the information processing apparatus 2. As shown in FIG. 3, theinformation processing apparatus 2 includes a processor 20, an inertialsensor 22, a DRAM 23, and a non-volatile memory 24 in addition to thedisplay section 21. The processor 20 includes a CPU and a GPU. The CPUcooperates with the DRAM 23 to perform a process described below, andthe GPU generates images (a left eye image and a right eye image)according to a command from the CPU. The generated left eye image andright eye image are displayed in the left area 21L and the right area21R, respectively, of the display section 21. The CPU and the GPU may bemounted on different chips, or may be mounted as an SoC(System-on-a-chip) on a single chip.

The inertial sensor 22 is a sensor for detecting the orientation of theinformation processing apparatus 2. Specifically, the inertial sensor 22includes an angular velocity sensor and an acceleration sensor. Theangular velocity sensor detects angular velocities about predeterminedthree axes (e.g., XYZ axes shown in FIG. 2). The acceleration sensordetects accelerations about predetermined three axes (e.g., the XYZ axesshown in FIG. 2).

The non-volatile memory 24 is a storage device that stores a program forperforming a process described below. The non-volatile memory 24 may be,for example, a flash memory. The non-volatile memory 24 may be anystorage device such as a magnetic disk or an optical disc.

FIG. 4 is a diagram showing the state where the user uses the goggleapparatus 10. As shown in FIG. 4, the user looks into the displaysection 21 by holding the goggle apparatus 10 to which the informationprocessing apparatus 2 is attached. If the user directs their face orthe entirety of their body in a left-right direction (the horizontaldirection or a “yaw direction”) or an up-down direction (the verticaldirection or a “pitch direction”) in this state, the orientation of thegoggle apparatus 10 (the information processing apparatus 2) changesfrom a reference orientation. The reference orientation is, for example,the orientation of the goggle apparatus 10 when the user faces thefront. For example, the reference orientation may be the orientation inwhich a down direction (a negative Y-axis direction in FIG. 2) along thedisplay section 21 (the bottom surface 15) of the goggle apparatus 10 isparallel to the direction of gravity.

Based on angular velocity values and/or acceleration values detected bythe inertial sensor 22, the information processing apparatus 2calculates the orientation of the information processing apparatus 2(the goggle apparatus 10). In accordance with the orientation of theinformation processing apparatus 2 (the goggle apparatus 10) in realspace, the orientations of the left virtual camera and the right virtualcamera in the virtual space are controlled.

In the virtual space, a virtual object is placed, and the user can viewa stereoscopic image of the virtual space including the virtual object.In the exemplary embodiment, in the virtual space, a user interface (UI)object is placed as an example of the virtual object.

FIG. 5 is a diagram showing an example of a virtual space VS. As shownin FIG. 5, in a virtual space VS, an xyz orthogonal coordinate system isset. An x-axis is an axis in a horizontal direction in the virtual spaceVS. A y-axis is an axis in a height direction in the virtual space VS. Az-axis is an axis perpendicular to the x-axis and the y-axis and is alsoan axis in a depth direction in the virtual space.

In the virtual space VS, a left virtual camera VCL and a right virtualcamera VCR are placed. The left virtual camera VCL and the right virtualcamera VCR are placed in the virtual space at a distance similar to thedistance between the left and right eyes of an average user. A left eyeimage and a right eye image obtained by viewing the virtual space fromthe left virtual camera VCL and the right virtual camera VCR are viewedby the left eye and the right eye, respectively, of the user, wherebythe user views a stereoscopic image of the virtual space. Hereinafter,the left virtual camera VCL and the right virtual camera VCR willoccasionally be collectively referred to as a “virtual camera VC”.

In the virtual space VS, a UI object 30 is placed. The UI object 30 is avirtual object that is operated by the user and displays information tobe presented to the user. For example, the UI object 30 provides a menufunction to the user. For example, the UI object 30 is displayed duringthe execution of a predetermined game application, and the user canperform a predetermined operation in a game using the UI object 30. TheUI object 30 is, for example, a plate-like object and is placed at aposition at a predetermined height in the virtual space and in theorientation in which the UI object 30 is perpendicular to an xz plane.

The UI object 30 includes icons 31 to 34 that can be selected by theuser. For example, the icon 34 selected by the user may be displayed ina different form from those of the other icons. The user selects any ofthe icons 31 to 34 by any method. For example, when a pointer 37 (seeFIG. 7A) is displayed at a predetermined position in a stereoscopicimage (e.g., the center of the image) displayed on the display section21, and the pointer is present in the display area of an icon, the iconmay be processed on the assumption that the icon is selected by theuser. In this case, to select a desired icon, the user changes theorientation of the goggle apparatus 10 so that the pointer enters thedisplay area of the desired icon. For example, using a handheldoperation apparatus connected to the information processing apparatus 2in a wireless or wired manner, the user may control the position of thepointer displayed in the stereoscopic image. For example, the operationapparatus may include a direction indication section that allows theuser to indicate a direction (e.g., an analog stick, a directional pad,a button, or the like), and based on an operation on the directionindication section, the position of the pointer may be controlled.

Each icon may be, for example, an icon for determining an operation inthe game, or may be an icon for using a predetermined item. Each iconmay be an icon for performing an operation on a menu screen provided bythe UI object 30 (e.g., the operation of returning to the previousscreen, or the operation of making a determination). Each icon may be anicon for selecting and starting a predetermined application.

The UI object 30 includes an information display object 35. Theinformation display object 35 is an object for providing information tothe user, using a character, an image, or the like.

In the virtual space, not only a game but also any other activity may beperformed. For example, a particular work (e.g., the work of flying anairplane or driving a vehicle) may be simulated. The function of eachicon included in the UI object 30 (an operation performed using the UIobject 30) differs depending on the activity performed in the virtualspace.

(Control of Movement of UI Object 30)

Next, a description is given of control of the UI object 30 when theuser directs the bottom surface 15 of the goggle apparatus 10 in theleft-right direction (the yaw direction) or the up-down direction (thepitch direction) while viewing the display section 21 of the goggleapparatus 10.

FIG. 6A is a diagram showing the virtual space viewed from above whenthe goggle apparatus 10 is maintained in the reference orientation. FIG.6B is a diagram showing the virtual space viewed from above when thegoggle apparatus 10 is rotated in the yaw direction (the left direction)from the reference orientation. FIG. 6C is a diagram showing the virtualspace viewed from above when the goggle apparatus 10 is further rotatedin the yaw direction (the left direction) from the state in FIG. 6B.

FIG. 7A is a diagram showing an example of an image viewed by the user,and also an example of an image of the virtual space viewed from thevirtual camera VC (VCL or VCR) in the orientation shown in FIG. 6A. FIG.7B is a diagram showing an example of an image viewed by the user, andalso an example of an image of the virtual space viewed from the virtualcamera VC in the orientation shown in FIG. 6B. FIG. 7C is a diagramshowing an example of an image viewed by the user, and also an exampleof an image of the virtual space viewed from the virtual camera VC inthe orientation shown in FIG. 6C. Although FIGS. 7A to 7C show planarrectangular images, the images shown in FIGS. 7A to 7C are actuallystereoscopic images and approximately circular similarly to the leftarea 21L and the right area 21R.

In the virtual camera VC, a camera coordinate system (an XYZ coordinatesystem) fixed to the virtual camera VC is set. An X-axis is an axis in aright direction of the virtual camera VC, a Y-axis is an axis in an updirection of the virtual camera VC, and a Z-axis is an axis in thedirection of the line of sight of the virtual camera VC. As shown inFIG. 6A, when the goggle apparatus 10 (the information processingapparatus 2) is maintained in the reference orientation, the virtualcamera VC is directed to the UI object 30.

In a state as shown in FIG. 6A, an image as shown in FIG. 7A isdisplayed on the display section 21. The user views a left eye image anda right eye image displayed on the display section 21, thereby viewing astereoscopic image as shown in FIG. 7A. As shown in FIG. 7A, the UIobject 30 is displayed at the front, and the background of the virtualspace is displayed behind the UI object 30. For example, a pointer 37 isset and displayed at the position of the center of the stereoscopicimage. The user controls the orientation of the goggle apparatus 10 sothat the pointer 37 is located in the display area of the UI object 30.Then, using the pointer 37, the user selects the icons 31 to 34 that canbe selected in the UI object 30. The pointer 37 may not be displayed onthe screen, and may be internally set in the information processingapparatus 2.

If the goggle apparatus 10 is rotated in the yaw direction (e.g., theleft direction) in real space, as shown in FIG. 6B, the virtual cameraVC also rotates in the yaw direction (e.g., the left direction) in thevirtual space. At this time, an image as shown in FIG. 7B is displayed.Specifically, when the goggle apparatus 10 is rotated in the leftdirection, the UI object 30 is displayed located to the right side ofthe front, and a part of the UI object 30 is no longer displayed.

If the goggle apparatus 10 is further rotated in the left direction fromthe state shown in FIG. 6B, as shown in FIG. 6C, the virtual camera VCfurther rotates in the left direction. At this time, an image as shownin FIG. 7C is displayed on the display section 21. Specifically, whenthe goggle apparatus 10 is further rotated in the left direction fromthe state shown in FIG. 6B without reversing the goggle apparatus 10 inthe right direction, the UI object 30 comes out of the image capturingrange of the virtual camera VC and is no longer displayed. In this case,the background of the virtual space except for the UI object 30 isdisplayed.

If the user further rotates the goggle apparatus 10 in the leftdirection from the state in shown in FIG. 6C and thereby can view thevirtual space further in the left direction.

Here, for example, when the user reverses the goggle apparatus 10 by apredetermined angle in the right direction in the orientation shown inFIG. 6C, the UI object 30 moves so that the UI object 30 is located infront of the virtual camera VC.

FIG. 8 is a diagram showing the state where the UI object 30 moves tothe front of the virtual camera VC when the goggle apparatus 10 isreversed by the predetermined angle in the right direction in theorientation shown in FIG. 6C. In FIG. 8, 30′ indicated by a dashed linerepresents the UI object before the movement, and 30 indicated by asolid line represents the UI object after the movement.

As shown in FIG. 8, for example, when the goggle apparatus 10 reversesby the predetermined angle in the right direction from the state wherethe goggle apparatus 10 is rotated in the left direction, the UI object30 moves to the front (the front in the left-right direction) of thevirtual camera VC. For example, the UI object 30 moves in the leftdirection along a circle centered at an intermediate position betweenthe left virtual camera VCL and the right virtual camera VCR whilechanging its direction. That is, the UI object 30 moves in the leftdirection on a plane parallel to the xz plane while maintaining thedistance from the virtual camera VC. The UI object 30 moves whilechanging its direction so that the UI object 30 is directed to thevirtual camera VC. Thus, when the virtual camera VC does not rotate inthe pitch direction in the virtual space, the line of sight of thevirtual camera VC is perpendicular to the UI object 30 after themovement. If the UI object 30 moves to the front of the virtual cameraVC, a stereoscopic image as shown in FIG. 7A is displayed on the displaysection 21.

The UI object 30 moves by a predetermined distance per frame time (e.g.,1/60 seconds). Thus, for example, when the UI object 30 reverses by thepredetermined angle in the right direction from the state where thegoggle apparatus 10 is rotated in the left direction, the UI object 30does not instantaneously move to the front of the virtual camera VC, butspends some time (e.g., several frame times to several tens of frametimes) moving. The UI object 30 may be instantaneously (in one frametime) moved to the front of the virtual camera VC.

Even when a part of the UI object 30 is displayed and the other part isnot displayed as shown in FIG. 7B, and if the goggle apparatus 10reverses by the predetermined angle in the right direction, the UIobject 30 may be moved to the front of the virtual camera VC. Even whenthe UI object 30 is displayed at the end of the screen and the entiretyof the UI object 30 is displayed, and if the goggle apparatus 10reverses by the predetermined angle in the right direction, the UIobject 30 may be moved to the front of the virtual camera VC.

When the pointer 37 is located in the display area of the UI object 30,the UI object 30 may not be moved. In a case where the pointer 37 is notlocated in the display area of the UI object 30, and when the goggleapparatus 10 reverses by the predetermined angle, the UI object 30 maybe moved.

As described above, in the exemplary embodiment, the UI object 30 is notmoved to the front of the virtual camera VC only by the virtual cameraVC rotating in one direction (e.g., the left direction) in the yawdirection and, for example, the UI object 30 coming out of the imagecapturing range of the virtual camera VC. After the virtual camera VCrotates in one direction (e.g., the left direction) in the yawdirection, and when the virtual camera VC rotates (reverses) by thepredetermined angle in the other direction (e.g., the right direction)in the yaw direction, the UI object 30 is moved to the front of thevirtual camera VC. The movement of the UI object 30 will be described indetail below.

(Details of Movement of UI Object 30)

FIG. 9 is a diagram showing examples of areas to which the UI object 30is moved. FIG. 9 shows a diagram where the virtual space is conceptuallydivided into a plurality of areas based on the virtual camera VC whenthe UI object 30 is located in front of the virtual camera VC.

In the exemplary embodiment, a process regarding the movement of the UIobject 30 differs depending on which of a “UI area”, a “B-area”, and an“A-area” shown in FIG. 9 the line of sight of the virtual camera VC ispresent in.

When the line of sight of the virtual camera VC is present in the “UIarea” shown in FIG. 9, the UI object 30 is displayed on the displaysection 21, and the UI object 30 does not move in the virtual space inaccordance with a change in the orientation of the goggle apparatus 10.The “UI area” is an area based on the UI object 30 and is also an areacorresponding to the UI object 30. Specifically, when the UI object 30is located in front of the virtual camera VC (front in the left-rightdirection and the up-down direction), the angle of the line of sight ofthe virtual camera VC is defined as 0 degrees. When the rotationalangles of the line of sight of the virtual camera VC in the yawdirection and the pitch direction in the virtual space are less than orequal to a predetermined threshold, it is determined that the line ofsight of the virtual camera VC is present in the “UI area” shown in FIG.9. More specifically, when the absolute value of the rotational angle ofthe line of sight of the virtual camera VC in the yaw direction is lessthan a first threshold (Ty), and the absolute value of the rotationalangle of the line of sight of the virtual camera VC in the pitchdirection is less than a second threshold (Tx), it is determined thatthe line of sight of the virtual camera VC is present in the “UI area”shown in FIG. 9.

Here, the UI area is determined based on the first threshold and thesecond threshold. For example, the UI area may coincide with the displayarea of the UI object 30. In this case, for example, when the rotationalangle of the line of sight of the virtual camera VC in the yaw directionreaches the first threshold (Ty), as shown in FIG. 7C, the entirety ofthe UI object 30 is no longer displayed. The UI area may be smaller thanthe display area of the UI object 30. In this case, when the rotationalangle of the line of sight of the virtual camera VC in the yaw directionis the first threshold (Ty), for example, as shown in FIG. 7B, a part ofthe UI object 30 is displayed. Even when the rotational angle of theline of sight of the virtual camera VC in the yaw direction is the firstthreshold (Ty), the entirety of the UI object 30 may be displayed. TheUI area determined based on the first threshold and the second thresholdmay be larger than the display area of the UI object 30.

The UI area may be such an area that the pointer 37 in a stereoscopicimage is located in the display area of the UI object 30. That is, theUI area may be an area that, when the pointer 37 moves along the outerperiphery of the display area of the UI object 30, is formed by thetrajectory of the movement.

When the line of sight of the virtual camera VC is present in the B-areashown in FIG. 9, a “movement process in the B-area” is performed. Thatis, when the absolute value of the rotational angle of the line of sightof the virtual camera VC in the yaw direction is greater than or equalto the first threshold (Ty), and the absolute value of the rotationalangle of the line of sight of the virtual camera VC in the pitchdirection is less than the second threshold (Tx), the “movement processin the B-area” is performed. In the “movement process in the B-area”,“the determination of whether or not the UI object 30 is to be moved” ismade. With reference to FIGS. 10 and 11, this determination isspecifically described below.

FIG. 10 is a diagram showing the virtual camera VC viewed from above thevirtual space and is also a diagram showing an example of the statewhere the virtual camera VC is rotated in the left direction and thenrotated in the right direction. FIG. 11 is a diagram showing the virtualcamera VC viewed from the horizontal direction in the virtual space andis also a diagram showing an example of the state where the virtualcamera VC is rotated in the down direction and then rotated in the updirection.

As shown in FIG. 10, for example, when the line of sight of the virtualcamera VC rotates by exceeding the first threshold “Ty” in the leftdirection (a positive direction) in the yaw direction, the line of sightof the virtual camera VC enters the B-area. After the line of sight ofthe virtual camera VC enters the B-area, the amount of change “y1” inthe left direction and the amount of change “y2” in the right directionare calculated. The amount of change “y1” in the left direction and theamount of change “y2” in the right direction are the relative amount ofrotation after the line of sight of the virtual camera VC enters theB-area. When the line of sight of the virtual camera repeatedly moves inthe left direction and the right direction, each of the amount of change“y1” in the left direction and the amount of change “y2” in the rightdirection is accumulated. For example, after the line of sight of thevirtual camera VC enters the B-area, and when the line of sight of thevirtual camera VC rotates “10” degrees in the left direction, “2”degrees in the right direction, “5” degrees in the left direction, and“3” degrees in the right direction in this order, the accumulation valueof the amount of change “y1” in the left direction is “10+5=15” degrees,and the accumulation value of the amount of change “y2” in the rightdirection is “2+3=5” degrees. The smaller value (the accumulation valueof the amount of change y2 in the right direction) of the accumulationvalue of the amount of change “y1” in the left direction and theaccumulation value of the amount of change “y2” in the right directionis stored as the amount of change Ry in the left-right direction. Inthis case, the accumulation value of the amount of change “y2” in theright direction, namely “5” degrees, is stored as the amount of changeRy in the left-right direction.

The same applies to rotation in the pitch direction. As shown in FIG.11, after the line of sight of the virtual camera VC enters the B-area(i.e., after the line of sight of the virtual camera VC rotates by anangle greater than or equal to the first threshold in the yawdirection), the line of sight of the virtual camera VC rotates “x1”degrees in the down direction (a negative direction) in the pitchdirection and then rotates “x2 (<x1)” degrees in the up direction.

FIG. 11 shows a case where the rotational angle in the pitch directionin the virtual space when the line of sight of the virtual camera VCenters the B-area is “0” degrees. There is also a case where, at thetime when the line of sight of the virtual camera VC enters the B-area,the line of sight of the virtual camera VC rotates by a predeterminedangle in the pitch direction in the virtual space. In this case, theamount of change in the down direction from the time when the line ofsight of the virtual camera VC enters the B-area is “x1”.

The smaller value (the accumulation value of the amount of change x2 inthe up direction) of the accumulation value of the amount of change “x1”in the down direction and the accumulation value of the amount of change“x2” in the up direction is stored as the amount of change Rx in theup-down direction. For example, after the line of sight of the virtualcamera VC enters the B-area, and when the line of sight of the virtualcamera VC rotates “10” degrees in the down direction, “2” degrees in theup direction, “3” degrees in the down direction, and “1” degrees in theup direction in this order, the accumulation value of the amount ofchange “x1” in the down direction is “10+3=13” degrees, and theaccumulation value of the amount of change “x2” in the up direction is“2+1=3” degrees. The smaller value (the accumulation value of the amountof change x2 in the up direction) of the accumulation value of theamount of change “x1” in the down direction and the accumulation valueof the amount of change “x2” in the up direction is stored as the amountof change Rx in the up-down direction. In this case, the accumulationvalue of the amount of change “x2” in the up direction, namely “3”degrees, is stored as the amount of change Rx in the up-down direction.

Here, when the line of sight of the virtual camera VC continues torotate in one direction (e.g., the left direction in the yaw direction)with respect to the yaw direction or the pitch direction, the amount ofchange in the other direction (e.g., the right direction in the yawdirection) is “0”. Thus, when the smaller value of the accumulationvalues of the amount of rotation in the one direction and the amount ofrotation in the other direction exceeds “0” degrees, the rotation of theline of sight of the virtual camera VC changes from the one direction tothe other direction. In contrast, when the smaller value of theaccumulation values of the amount of rotation in one direction and theamount of rotation in the other direction is “0” degrees, this meansthat the line of sight of the virtual camera continues to rotate in theone direction, or the line of sight of the virtual camera does notrotate.

Thus, in the exemplary embodiment, based on the smaller values Rx andRy, it is detected whether or not the rotation of the line of sight ofthe virtual camera VC changes (i.e., reverses) from one direction to theother direction. When a change from one direction to the other directionis detected, the UI object 30 is moved to the front of the virtualcamera VC.

Specifically, when the sum of the amount of change Ry in the left-rightdirection and the amount of change Rx in the up-down direction isgreater than or equal to a predetermined value (e.g., “6” degrees), theUI object 30 is moved to the front of the virtual camera VC. On theother hand, when the sum is less than the predetermined value, the UIobject 30 is not moved. For example, in a case where the line of sightof the virtual camera VC is present in the B-area, and when the line ofsight of the virtual camera VC rotates “10 degrees in the leftdirection” →“3 degrees in the right direction” →“20 degrees in the leftdirection” in this order, the amount of change Ry in the left-rightdirection is “3” degrees, and the amount of change Rx in the up-downdirection is “0” degrees. In this case, since “Rx+Ry” is less than thepredetermined value, the UI object 30 is not moved to the front of thevirtual camera VC.

On the other hand, in a case where the line of sight of the virtualcamera VC is present in the B-area, and when the line of sight of thevirtual camera VC rotates “10 degrees in the left direction” →“3 degreesin the right direction” →“20 degrees in the left direction” →“3 degreesin the right direction” in this order, the amount of change Ry in theleft-right direction is “6” degrees. In this case, since “Rx+Ry” isgreater than or equal to the predetermined value, the UI object 30 ismoved to the front of the virtual camera VC. When the line of sight ofthe virtual camera VC rotates in an oblique direction, Rx and Rydescribed above are calculated by dividing the rotation direction into acomponent in the left-right direction (the yaw direction) and acomponent in the up-down direction (the pitch direction). For example,when the line of sight of the virtual camera VC rotates in the “left-updirection (10 degrees with the component in the left direction and 10degrees with the component in the up direction)” and next rotates in the“right-down direction (3 degrees with the component in the rightdirection and 3 degrees with the component in the down direction)”, Rxis “3” degrees, and Ry “3” degrees. In this case, “Rx+Ry” is greaterthan or equal to the predetermined value. Thus, the UI object 30 ismoved to the front of the virtual camera VC.

As described above, in the “movement process in the B-area”, it isdetected that the virtual camera VC rotates in one direction in the yawdirection or the pitch direction and then rotates in the oppositedirection. In a case where the virtual camera VC rotates in the oppositedirection, and when a value obtained by accumulating the rotationalangle in the opposite direction is greater than or equal to thepredetermined value, the UI object 30 moves to the front of the virtualcamera VC. That is, in a case where the line of sight of the virtualcamera VC changes from one direction to the other direction in apredetermined rotation direction (the yaw direction or the pitchdirection), and further, the rotational angle in the other direction isgreater than or equal to the predetermined value, the UI object 30 ismoved.

In a case where the line of sight of the virtual camera VC is present inthe B-area, and even if “Rx+Ry” is less than the predetermined value,and when the user performs a predetermined operation, the UI object 30may be moved to the front of the virtual camera VC. For example, when apredetermined button or place in the goggle apparatus 10 is pressed,tapped, or hit, the UI object 30 may be moved to the front of thevirtual camera VC. For example, when the acceleration sensor of theinertial sensor 22 of the goggle apparatus 10 detects an accelerationvalue greater than or equal to a predetermined value, the UI object 30may be moved to the front of the virtual camera VC.

Referring back to FIG. 9, when the line of sight of the virtual cameraVC is present in the A-area, i.e., when the absolute value of therotational angle of the line of sight of the virtual camera VC in thepitch direction exceeds the second threshold (Tx), a “movement processin the A-area” is performed. In the “movement process in the A-area”,the above determination in the B-area is not made, and the UI object 30is moved so that the UI object 30 is always located in front of thevirtual camera VC with respect to the left-right direction. Here, whenthe angle of depression or the angle of elevation of the virtual cameraVC (the goggle apparatus 10) exceeds the second threshold, the “movementprocess in the A-area” is performed. However, a threshold regarding theangle of depression and a threshold regarding the angle of elevation maynot need to be the same value, and may be different from each other.

FIG. 12 is a diagram showing the state where the UI object 30 moves in acase where the line of sight of the virtual camera VC is present in theA-area.

In FIG. 12, 30′ indicated by a dashed line represents the UI objectbefore the movement, and 30 indicated by a solid line represents the UIobject after the movement. As shown in FIG. 12, for example, when theline of sight of the virtual camera VC is present in the A-area abovethe UI object 30, the UI object 30 moves so that the UI object 30 islocated in front of the virtual camera VC with respect to the left-rightdirection. The rotational angle of the virtual camera VC in the yawdirection (the left-right direction) based on the UI object 30 after themovement is “0” degrees. That is, based on the direction of the line ofsight of the virtual camera VC, the UI object 30 after the movement isnot shifted in the left-right direction. Thus, when the line of sight ofthe virtual camera VC is rotated in the down direction (when the virtualcamera VC is rotated so that the rotational angle of the virtual cameraVC in the pitch direction is “0” degrees), the UI object 30 is locatedin front of the virtual camera VC.

Here, a case is assumed where, after the line of sight of the virtualcamera VC enters the B-area, the line of sight of the virtual camera VCenters the A-area without the UI object 30 moving by the “movementprocess in the B-area”. In this case, when the line of sight of thevirtual camera VC enters the A-area, the UI object 30 starts moving bythe “movement process in the A-area”. In this case, the UI object 30 mayspend some time (e.g., several frame times to several tens of frametimes) moving or instantaneously (in one frame time) move to the frontof the virtual camera VC with respect to the left-right direction.

When the line of sight of the virtual camera VC enters the A-area fromthe state where the line of sight of the virtual camera VC is present inthe “UI area”, the UI object 30 moves by the “movement process in theA-area”. When the line of sight of the virtual camera VC is present inthe A-area, and if the virtual camera VC rotates in the left-rightdirection, the UI object 30 moves in accordance with the rotation of thevirtual camera VC in the left-right direction. That is, while thevirtual camera rotates in the left-right direction, the UI object 30always moves following the rotation of the virtual camera in theleft-right direction.

FIG. 13A is a diagram showing an example of an image displayed on thedisplay section 21 in a case where the line of sight of the virtualcamera VC rotates in the down direction and enters the A-area. FIG. 13Bis a diagram showing an example of an image displayed on the displaysection 21 when the virtual camera VC rotates in the left direction fromthe state in FIG. 13A.

As shown in FIG. 13A, when the line of sight of the virtual camera VCrotates in the down direction (i.e., when the goggle apparatus 10rotates in the down direction), a part of the lower side of the UIobject 30 is displayed in an upper area of the screen. The UI object 30is displayed at the center of the screen with respect to the left-rightdirection. In the background of the virtual space, for example, avirtual object 40 is present. The virtual object 40 is displayed at theleft end of the screen.

When the line of sight of the virtual camera VC rotates in the leftdirection in this state (i.e., when the goggle apparatus 10 rotates inthe left direction), as shown in FIG. 13B, the position of the UI object30 in the left-right direction does not change. This is because the UIobject 30 moves in the virtual space in accordance with the rotation ofthe virtual camera VC in the yaw direction by the above movement processin the A-area. Meanwhile, the virtual object 40 present in thebackground of the virtual space is displayed on the right side of itsposition in FIG. 13A. While the user continues to rotate the goggleapparatus 10 in the left direction, the position of the UI object 30 onthe screen does not change, and the background of the virtual spacemoves in the right direction.

The sizes and the shapes of the “UI area”, the “A-area”, and the“B-area” shown in FIG. 9 may differ in accordance with the size and theshape of the UI object 30 to be displayed. The UI object 30 may be of aplurality of types, and the size and the shape of each area may differin accordance with the type of the UI object 30. The size and the shapeof each area may differ in accordance with the scene of the virtualspace (the scene of the background). In this case, the first threshold(Ty) and the second threshold (Tx) described above differ in accordancewith the size and the shape of each area. The size of the “UI area” maynot necessarily need to match the size of the UI object 30. For example,there may be a margin around the UI object 30, and an area including anarea where the UI object 30 is displayed and the margin around the UIobject 30 may be set as the “UI area”. Conversely, the “UI area” may besmaller than the area where the UI object 30 is displayed.

As described above, in the exemplary embodiment, when the rotationalangles of the virtual camera VC in the yaw direction and the pitchdirection (the rotational angles of the goggle apparatus 10 in the yawdirection and the pitch direction) in the virtual space exceed thepredetermined threshold, a movement process for moving the UI object 30is performed.

Specifically, when the absolute value of the rotational angle of thevirtual camera VC in the yaw direction in the virtual space is greaterthan the first threshold (Ty), and the absolute value of the rotationalangle in the pitch direction is smaller than the second threshold (Tx)(when the line of sight of the virtual camera VC is present in theB-area), the “movement process in the B-area” is performed. In the“movement process in the B-area”, based on the rotational angles of thevirtual camera VC in the yaw direction and the pitch direction, thedetermination of whether or not the UI object 30 is to be moved is made.Specifically, when the rotation of the virtual camera VC in the yawdirection changes from one direction to the other direction, the valueRy obtained by accumulating the rotational angle in the other directionis calculated. When the rotation of the virtual camera VC in the pitchdirection changes from one direction to the other direction, the valueRx obtained by accumulating the rotational angle in the other directionis calculated. Then, the total of Rx and Ry is greater than or equal tothe predetermined value, it is determined that the UI object 30 is to bemoved. Then, the UI object 30 is moved to the front of the virtualcamera VC.

The user may wish to view the UI object 30 in the virtual space, or maywish to view an object in a portion of the virtual space (the backgroundor another object in the virtual space) different from a portion of thevirtual space where the UI object 30 is located. When the line of sightof the virtual camera VC (the orientation of the goggle apparatus 10)continues to rotate in one direction, it is considered that the userwishes to view the portion of the virtual space different from theportion of the virtual space where the UI object 30 is located. Thus, inthe exemplary embodiment, when the line of sight of the virtual cameraVC continues to rotate in one direction (e.g., the left direction withrespect to the yaw direction, or the up direction with respect to thepitch direction) or is at rest, the UI object 30 is not moved to thefront of the virtual camera VC. Thus, the user continues to rotate thegoggle apparatus 10, for example, in the left direction and thereby canlook over the portion of the virtual space other than the portion of thevirtual space where the UI object 30 is located.

On the other hand, when the line of sight of the virtual camera VCreverses from a direction away from the UI object 30 to a directiontoward the UI object 30, there is a possibility that the user wishes toview the UI object 30 rather than another object in the virtual space.Thus, in the exemplary embodiment, in a case where the line of sight ofthe virtual camera VC reverses from one direction (a direction away fromthe UI object 30) to the other direction, and further, when the amountof rotation in the other direction becomes greater than or equal to apredetermined value, the UI object 30 is moved to the front of thevirtual camera VC.

By performing such control, in the exemplary embodiment, it is possibleto allow the user to view a portion of the virtual space different froma portion of the virtual space where the UI object 30 is located, andalso move the UI object 30 present in the virtual space to the front sothat the UI object 30 is easily operated (viewed).

For example, it is possible that at the time when the virtual camera VCis directed in a direction different from that of the UI object 30, andthe UI object 30 comes out of the image capturing range of the virtualcamera VC, the UI object 30 is moved to the front of the virtual cameraVC. In this case, however, at the time when the UI object 30 comes outof the image capturing range of the virtual camera VC, the UI object 30moves to the front of the user. Thus, the user cannot view the portionof the virtual space other than the portion of the virtual space wherethe UI object 30 is located.

In contrast, in the exemplary embodiment, when the goggle apparatus 10continues to be rotated in one direction (a direction away from the UIobject 30) in the yaw direction or is at rest, the UI object 30 does notmove to the front of the virtual camera VC. Thus, the user can view theportion of the virtual space other than the portion of the virtual spacewhere the UI object 30 is located. If the user wishes to view or operatethe UI object 30 when looking over the virtual space, for example, theuser reverses the goggle apparatus 10 to the other direction in the yawdirection or swings up and down the goggle apparatus 10, and thereby canmove the UI object 30 to the front of the user. Consequently, forexample, it is possible to easily select an icon in the UI object 30using a pointer and improve the operability of the UI object 30 in a VRspace. In a case where at least a part of the UI object 30 is displayed,and even when the UI object 30 is displayed at the end of the screen, itis possible to move the UI object 30 to the front by merely reversingthe goggle apparatus 10 by a predetermined angle. Thus, it is possibleto improve the operability.

In the exemplary embodiment, the UI object 30 is moved in the virtualspace. Thus, it may be possible to reduce VR sickness.

In the exemplary embodiment, when the rotational angle of the virtualcamera VC in the pitch direction in the virtual space is greater thanthe second threshold (Tx) (when the line of sight of the virtual cameraVC is present in the A-area), the “movement process in the A-area” isperformed. In the “movement process in the A-area”, in accordance withthe rotation of the virtual camera VC in the yaw direction, the UIobject 30 always moves to the front of the virtual camera VC (the frontwith respect to the left-right direction). Thus, after the line of sightof the virtual camera VC enters the A-area, and when the user views thedisplay section 21 with the goggle apparatus 10 in the referenceorientation, the UI object 30 is always located in front of the user.For example, after the user places the goggle apparatus 10 on a surfacein real space (a surface parallel to the ground, e.g., a table) so thatthe bottom surface 15 of the goggle apparatus 10 faces down or up, andwhen the user views the display section 21 again by holding the goggleapparatus 10, the UI object 30 always seems to be located in front ofthe user.

For example, it is assumed that two users experience VR using a singlegoggle apparatus 10. One of the users experiences VR using the goggleapparatus 10 and then passes the goggle apparatus 10 to the other user.At this time, the orientation of the goggle apparatus 10 changes, forexample, so that the bottom surface 15 faces down. In this case, if theUI object 30 does not move by the above movement process in the A-area,and when the other user views the display section 21 by holding thegoggle apparatus 10, the UI object 30 is not necessarily present infront of the other user. If the UI object 30 does not move by themovement process in the A-area, the UI object 30 maintains the positionwhen the goggle apparatus 10 is passed to the other user. Thus, when theother user views the display section 21 of the goggle apparatus 10, theUI object 30 may be located behind the other user, or may be located inthe right direction. In the exemplary embodiment, however, the UI object30 moves by the movement process in the A-area. Thus, when the otheruser views the display section 21 of the goggle apparatus 10, the UIobject 30 is located in front of the other user. Thus, the operabilityfor the other user is improved.

(Details of Processing) Next, an example of the processing performed bythe information processing apparatus 2 is specifically described. First,data stored in the information processing apparatus 2 is described.

FIG. 14 is a diagram showing an example of data stored in (the DRAM 23of) the information processing apparatus 2. As shown in FIG. 14, theinformation processing apparatus 2 stores a predetermined program,angular velocity data, virtual camera data, UI data, accumulation valuesX1 and X2, accumulation values Y1 and Y2, the amount of change Rx, andthe amount of change Ry. In addition to these pieces of data, variouspieces of data such as data regarding an object placed in the virtualspace and operation data corresponding to an operation of the user arestored.

The program is a program for executing the processing described below.The program is stored, for example, in the non-volatile memory 24 or anexternal storage medium and loaded from the non-volatile memory 24 orthe external storage medium into the DRAM 23. The program may beacquired from another apparatus via a network (e.g., a LAN, a WAN, theInternet, or the like).

The angular velocity data is data regarding angular velocities outputfrom the inertial sensor 22.

The virtual camera data is data regarding the positions and theorientations of the left virtual camera VCL and the right virtual cameraVCR.

The UI data is data regarding the position and the type of the UI object30, the icons 31 to 34 included in the UI object 30, the informationdisplay object 35, and the like.

The accumulation value X1 is a value used in the movement process in theB-area and is also a value obtained by accumulating the amount of changex1 in the down direction (see FIG. 11). The accumulation value X2 is avalue used in the movement process in the B-area and is also a valueobtained by accumulating the amount of change x2 in the up direction(see FIG. 11).

The accumulation value Y1 is a value used in the movement process in theB-area and is also a value obtained by accumulating the amount of changey1 in the left direction (see FIG. 10). The accumulation value Y2 is avalue used in the movement process in the B-area and is also a valueobtained by accumulating the amount of change y2 in the right direction(see FIG. 10).

The amount of change Rx is a value used in the movement process in theB-area and is also the smaller value of the accumulation values X1 andX2 of the amounts of change in the line of sight of the virtual cameraVC in the up-down direction (the pitch direction). The amount of changeRy is a value used in the movement process in the B-area and is also thesmaller value of the accumulation values Y1 and Y2 of the amounts ofchange in the line of sight of the virtual camera VC in the left-rightdirection (the yaw direction). Rx and Ry are initially set to “0”.

(Description of Main Flow)

Next, the details of a main process performed by the informationprocessing apparatus 2 are described. FIG. 15 is a flow chart showing anexample of a main process performed by the processor 20 of theinformation processing apparatus 2. The processing shown in FIG. 15 isperformed by the CPU or the GPU of the information processing apparatus2 executing a predetermined program. FIG. 15 shows only processesregarding the movement of the UI object 30 described above and omitsother processes (e.g., processes corresponding to an action (operation)of the user performed in the virtual space).

As shown in FIG. 15, first, the processor 20 performs an initial process(step S100). In the initial process, first, initialization regarding theorientation of the goggle apparatus 10 (the information processingapparatus 2) is performed. For example, the user is instructed to placethe goggle apparatus 10 including the information processing apparatus 2on a table or the like, thereby initializing the orientation of thegoggle apparatus 10. In the initial process, the xyz coordinate systemis set in the virtual space, and the UI object 30, the left virtualcamera VCL, the right virtual camera VCR, the background, other objects,and the like are placed in the virtual space. After the process of stepS100, the processor 20 repeatedly executes the processes of steps S101to S109 every predetermined frame time (e.g., 1/60 seconds).

After the process of step S100, the processor 20 acquires data (e.g.,angular velocity values) from the inertial sensor 22 (step S101).

Next, based on the data acquired from the inertial sensor 22 in stepS101, the processor 20 performs an orientation calculation process (stepS102). Specifically, based on the angular velocity values from theinertial sensor 22, the processor 20 calculates the orientation of thegoggle apparatus 10 (the information processing apparatus 2). Theprocessor 20 integrates the angular velocity values from the inertialsensor 22, thereby calculating a change in the orientation of the goggleapparatus 10 from the initialization in step S100 and acquiring theorientation of the goggle apparatus 10. In accordance with thecalculated orientation of the goggle apparatus 10, the processor 20 setsthe orientation of the virtual camera VC (the left virtual camera VCLand the right virtual camera VCR) in the virtual space. For example, theprocessor 20 sets the orientation of the virtual camera VC so that theorientations of the left virtual camera VCL and the right virtual cameraVCR in the virtual space match the orientation of the goggle apparatus10 in real space. Consequently, for example, when the goggle apparatus10 rotates 5 degrees in the left direction, the left virtual camera VCLand the right virtual camera VCR also rotate 5 degrees in the leftdirection.

Subsequently, based on the orientation calculated in step S102, theprocessor 20 determines whether or not the line of sight of therotational angle of the virtual camera VC in the pitch direction isgreater than or equal to the second threshold (Tx) (step S103). When itis determined that the line of sight of the rotational angle of thevirtual camera VC in the pitch direction is greater than or equal to thesecond threshold (Tx) (step S103: YES), the processor 20 performs themovement process in the A-area (step S104). When the process of stepS104 is executed, the processor 20 executes the process of step S108next.

Specifically, in step S104, the processor 20 moves the UI object 30 tothe front of the virtual camera VC with respect to the left-rightdirection. The processor 20 moves the UI object 30 on a plane parallelto the xz plane in the virtual space while maintaining the height of theUI object 30 in the virtual space. For example, the processor 20 movesthe UI object 30 along a circle centered at the position of the virtualcamera VC (e.g., the intermediate position between the left virtualcamera VCL and the right virtual camera VCR) so that the UI object 30 islocated in front of the virtual camera VC with respect to the left-rightdirection. Consequently, the UI object 30 moves as shown in FIG. 12. Theprocessor 20 may move the UI object 30 by a predetermined distance inone frame time. In this case, the process of step S104 is repeatedlyexecuted every frame time, and the state where the UI object 30 moves tothe front of the virtual camera VC with respect to the left-rightdirection is displayed on the display section 21.

On the other hand, when it is determined that the line of sight of therotational angle of the virtual camera VC in the pitch direction is lessthan the second threshold (Tx) (step S103: NO), the processor 20determines whether or not the rotational angle of the line of sight ofthe virtual camera VC in the yaw direction is greater than or equal tothe first threshold (Ty) (step S105).

When it is determined that the rotational angle of the line of sight ofthe virtual camera VC in the yaw direction is greater than or equal tothe first threshold (Ty) (step S105: YES), the processor 20 performs themovement process in the B-area (step S106). The details of the movementprocess in the B-area will be described below. When the process of stepS106 is executed, the processor 20 executes the process of step S108next.

When it is determined that the rotational angle of the line of sight ofthe virtual camera VC in the yaw direction is less than the firstthreshold (Ty) (step S105: NO), the processor 20 performs a UI process(step S107). In the UI process, a selection process for selecting eachicon in the UI object 30 and a process after the selection areperformed. For example, when the pointer 37 located at a predeterminedposition (e.g., the center) in a stereoscopic image is present in thedisplay area of the icon 31 in the UI object 30, the icon 31 isselected. Each icon is assigned a process in a case where the icon isselected (e.g., the switching of screens, an operation in a game, thestart of an application, or the like). The processor 20 performs theprocess corresponding to the selected icon. In step S107, the processor20 resets the amounts of change Rx and Ry and the accumulation valuesX1, X2, Y1, and Y2 calculated in step S106 described below to “0”. Whenthe process of step S107 is executed, the processor 20 executes theprocess of step S108 next.

In step S108, the processor 20 performs an image generation process.Specifically, the processor 20 captures the virtual space using the leftvirtual camera VCL, thereby generating a left eye image, and alsocaptures the virtual space using the right virtual camera VCR, therebygenerating a right eye image.

Then, the processor 20 displays on the display section 21 the left eyeimage and the right eye image generated in step S108 (step S109).Specifically, the processor 20 displays the left eye image in the leftarea 21L and displays the right eye image in the right area 21R. Whenthe process of step S109 is executed, the processor 20 executes theprocess of step S101 again.

(Movement Process in B-Area)

Next, the movement process in the B-area in the above step S106 isdescribed. FIG. 16 is a flow chart showing an example of the movementprocess in the B-area in step S106.

As shown in FIG. 16, first, the processor 20 calculates the amount ofchange x1 in the up direction and the amount of change x2 in the downdirection in the rotational angle of the line of sight of the virtualcamera VC in the pitch direction in the virtual space (step S130). Forexample, based on the orientation of the virtual camera VC in theprevious processing loop and the orientation of the virtual camera VC inthe current processing loop, the processor 20 determines whether or notthe line of sight changes in the up direction or the down direction,thereby calculating the amount of change x1 or x2 in the line of sightof the virtual camera VC. Then, the processor 20 adds the calculatedamount of change x1 or x2 to the accumulation value X1 or X2. Forexample, when the line of sight of the virtual camera VC continues tochange in the up direction, the accumulation value X1 increases, whilethe amount of change x2 in the down direction is “0” degrees, and theaccumulation value X2 of the amount of change x2 is also “0” degrees.For example, as in the example of FIG. 11, when the virtual camera VCrotates x1 degrees in the down direction and then rotates x2 degrees inthe up direction, “x1” is set as the accumulation value X1, and “x2” isset as the accumulation value X2.

Next, the processor 20 sets the smaller value of the accumulation valuesX1 and X2 updated in step S130 as the amount of change Rx in the up-downdirection (step S131).

Subsequently, the processor 20 calculates the amount of change y1 in theleft direction and the amount of change y2 in the right direction in therotational angle of the line of sight of the virtual camera VC in theyaw direction in the virtual space (step S132). For example, based onthe orientation of the virtual camera VC in the previous processing loopand the orientation of the virtual camera VC in the current processingloop, the processor 20 determines whether or not the line of sightchanges in the left direction or the right direction, therebycalculating the amount of change y1 or y2 in the line of sight of thevirtual camera VC. Then, the processor 20 adds the calculated amount ofchange y1 or y2 to the accumulation value Y1 or Y2. For example, whenthe line of sight of the virtual camera VC continues to change in theleft direction, the accumulation value Y1 increases, while the amount ofchange y2 in the right direction is “0” degrees, and the accumulationvalue Y2 of the amount of change y2 is also “0” degrees. For example, asin the example of FIG. 10, when the virtual camera VC rotates y1 degreesin the left direction and then rotates y2 degrees in the rightdirection, “y1” is set as the accumulation value Y1, and “y2” is set asthe accumulation value Y2.

Next, the processor 20 sets the smaller value of accumulation values Y1and Y2 updated in step S132 as the amount of change Ry in the left-rightdirection (step S133).

Subsequently, the processor 20 determines whether or not the sum of Rxset in step S131 and Ry set in step S133 is greater than or equal to apredetermined value (step S134).

When it is determined that “Rx+Ry” is greater than or equal to thepredetermined value (step S134: YES), the processor 20 moves the UIobject 30 to the front of the virtual camera VC (step S135).Specifically, the processor 20 moves the UI object 30 on a planeparallel to the xz plane while maintaining the height of the UI object30 in the virtual space. For example, the processor 20 moves the UIobject 30 along a circle centered at the position of the virtual cameraVC (e.g., the intermediate position between the left virtual camera VCLand the right virtual camera VCR) so that the UI object 30 is located infront of the virtual camera VC in the left-right direction.Consequently, the UI object 30 moves as shown in FIG. 8.

When the process of step S135 is executed, or when the determination isNO in step S134, the processor 20 determines whether or not apredetermined operation is performed (step S136). Here, the“predetermined operation” may be, for example, the operation ofpressing, tapping, or hitting a predetermined button or place in thegoggle apparatus 10.

When it is determined that the predetermined operation is performed(step S136: YES), the processor 20 executes the process of step S137.The process of step S137 is similar to the process of step S135.

When the process of step S137 is executed, or when the determination isNO in step S136, the processing of the processor 20 shown in FIG. 16ends, and the processing returns to FIG. 15.

The processes shown in the above flow charts are merely illustrative,and the order and the contents of the processes may be appropriatelychanged. A threshold may or may not be included in a determination. Forexample, “the determination of whether or not a certain value is greaterthan or equal to a threshold” may be replaced by “the determination ofwhether or not a certain value is greater than a threshold”.

As described above, when the line of sight of the virtual camera VC ispresent in the B-area, in the movement process in the B-area, it isdetected that the line of sight of the virtual camera VC changes fromone direction to the other direction with respect to the yaw directionor the pitch direction. Specifically, when the value of Rx or Ry set instep S133 or S131 exceeds “0” degrees, the line of sight of the virtualcamera VC changes from one direction to the other direction. In a casewhere the line of sight of the virtual camera VC changes from onedirection to the other direction, and when a value obtained byaccumulating the rotational angle in the other direction (Rx and/or Ry)is greater than or equal to a predetermined value, the UI object 30 ismoved to the front of the virtual camera VC. Consequently, the user canlook over a portion of the virtual space other than a portion of thevirtual space where the UI object 30 is located. Further, when the userwishes to view the UI object 30, the user can move the UI object 30 tothe front.

When the line of sight of the virtual camera VC is present in theA-area, in the movement process in the A-area, the UI object 30 is movedso that the UI object 30 is always located in front of the virtualcamera VC with respect to the left-right direction. Consequently, forexample, by directing the goggle apparatus 10 in the up direction or thedown direction, it is possible to move the UI object 30 to the front.

(Variations)

While image processing according to the exemplary embodiment has beendescribed above, the exemplary embodiment is merely an example and canbe modified as follows, for example.

For example, in the above exemplary embodiment, in the movement processin the A-area and the movement process in the B-area, the UI object 30is moved so that the UI object 30 is located in front of the virtualcamera VC. In another exemplary embodiment, control (a change in theorientation and/or the movement) of the virtual camera VC may beperformed so that the UI object 30 is located in front of the virtualcamera VC. For example, the orientation of the virtual camera VC may bechanged or the virtual camera VC may be moved by fixing the UI object 30so that the UI object 30 is located in front of the virtual camera VC.In another exemplary embodiment, both the movement of the UI object 30and control of the virtual camera VC may be performed so that the UIobject 30 is located in front of the virtual camera VC. That is, atleast either one of the movement of the UI object 30 and control (achange in the orientation and/or the movement) of the virtual camera VCmay be performed so that the UI object 30 is located in front of thevirtual camera VC. When the movement of the UI object 30 is performed,as shown in the above exemplary embodiment, the UI object 30 may bemoved, and the UI object 30 may also be rotated so that the UI object 30is directed to the virtual camera VC.

In the above exemplary embodiment, when a condition is satisfied, the UIobject 30 is moved to the front of the virtual camera VC. In anotherexemplary embodiment, when a condition is satisfied, the UI object 30may be moved, or the virtual camera VC may be controlled (e.g., thevirtual camera may be moved and/or rotated), or both the movement of theUI object 30 and control of the virtual camera VC may be performed sothat at least a part of the UI object 30 is located in the imagecapturing range of the virtual camera VC.

In the above exemplary embodiment, in the movement process in theB-area, in a case where the rotation of the virtual camera VC in the yawdirection or the pitch direction changes from one direction to the otherdirection, it is determined whether or not the smaller value of a valueobtained by accumulating the rotational angle in the other direction anda value obtained by accumulating the rotational angle in the onedirection is greater than or equal to a predetermined value.Specifically, it is determined whether or not the sum of the amounts ofchange Ry and Rx is greater than or equal to a predetermined value. Thedetermination of whether or not the UI object 30 is to be moved may bemade by another method. For example, in another exemplary embodiment, itmay be determined whether or not either one of the above amounts ofchange Ry and Rx is greater than or equal to a predetermined value. In acase where the rotation of the virtual camera VC in the yaw direction orthe pitch direction changes from one direction to the other direction,and when a value obtained by accumulating the rotational angle in theother direction and/or a value obtained by accumulating the rotationalangle in the one direction are greater than or equal to a predeterminedvalue, the UI object 30 may be moved.

In another exemplary embodiment, the rotational angle in the otherdirection and the rotational angle in one direction may not beaccumulated, and when the rotational angle in the other direction and/orthe rotational angle in one direction are greater than or equal to apredetermined value, the UI object 30 may be moved. In this case, in acase where the rotation of the virtual camera VC in one direction andthe rotation of the virtual camera VC in the other direction arerepeated multiple times, and when the rotational angle in the otherdirection or the rotational angle in the one direction at each time isless than a predetermined value, the UI object 30 is not moved. On theother hand, in a case where the rotation of the virtual camera VC in onedirection and the rotation of the virtual camera VC in the otherdirection are repeated multiple times, and when the rotational angle inthe other direction or the rotational angle in the one direction at atime is greater than or equal to the predetermined value, the UI object30 is moved.

The determination of whether or not the amount of change Ry with respectto the yaw direction is greater than or equal to the predetermined valueand the determination of whether or not the amount of change Rx withrespect to the pitch direction is greater than or equal to thepredetermined value may be separately made. In this case, when eitherone of Rx and Ry is greater than or equal to the predetermined value,the UI object 30 is moved.

A threshold may be varied between the amount of change Ry with respectto the yaw direction and the amount of change Rx with respect to thepitch direction. For example, regarding the yaw direction, when Ry isgreater than or equal to a first predetermined value (e.g., 4 to 8degrees), the UI object 30 may be moved. Regarding the pitch direction,when Rx is greater than or equal to a second predetermined value (e.g.,10 to 20 degrees) greater than the first predetermined value, the UIobject 30 may be moved. These thresholds may be varied in accordancewith the type of the UI object 30 to be displayed, or the scene of thevirtual space (the scene of the background).

In the above exemplary embodiment, under a first condition that the lineof sight of the virtual camera VC reverses from one direction to theother direction with respect to the yaw direction and the pitchdirection, in a case where the first condition is satisfied, and when asecond condition is further satisfied (specifically, when theaccumulation value of the rotational angle in the other direction isgreater than or equal to a predetermined value), the UI object 30 ismoved to the front of the virtual camera VC. In another exemplaryembodiment, when the first condition is satisfied, i.e., when the lineof sight of the virtual camera VC reverses from one direction to theother direction, the UI object 30 may be moved to the front of thevirtual camera VC. In this case, at the moment when the line of sight ofthe virtual camera VC reverses, the UI object 30 moves to the front ofthe virtual camera VC, regardless of the accumulation values.

In another exemplary embodiment, when the number of times the line ofsight of the virtual camera VC reverses is greater than or equal to apredetermined number of times, the UI object 30 may be moved to thefront of the virtual camera VC. For example, in a case where the virtualcamera VC rotates in the left direction, enters the B-area, and thenrepeats rotating in the right direction and the left direction, and whenthe number of times the virtual camera VC reverses (reverses in theright direction and reverses in the left direction) is greater than orequal to a predetermined number of times, the UI object 30 may be moved.

In another exemplary embodiment, when the line of sight of the virtualcamera VC reverses from one direction (e.g., the left direction) to theother direction (e.g., the right direction), based on the angle in theother direction (the right direction) and the one direction (the leftdirection) after the reversal, the UI object 30 may be moved.

That is, in the movement process in the B-area, based on the reversal ofthe line of sight of the virtual camera VC from one direction to theother direction, the UI object 30 may be moved to the front of thevirtual camera VC. Here, “based on the reversal of the line of sight ofthe virtual camera VC from one direction to the other direction”includes the following cases, for example.

-   -   In a case where the line of sight of the virtual camera VC        reverses from one direction to the other direction in a        predetermined rotation direction, the rotational angle in the        other direction or the accumulation value (Rx or Ry described        above) of the rotational angle is greater than or equal to a        predetermined value.    -   The number of times the line of sight of the virtual camera VC        reverses in a predetermined rotation direction is a        predetermined value.    -   The line of sight of the virtual camera VC changes from one        direction to the other direction in a predetermined rotation        direction.

In the above exemplary embodiment, based on the angle of the line ofsight of the virtual camera VC, it is determined which of the “A-area”and the “B-area” shown in FIG. 9 the line of sight of the virtual cameraVC is present in. Then, the UI object 30 is moved in accordance witheach area. In another exemplary embodiment, instead of the angle of theline of sight of the virtual camera VC, for example, in accordance withwhich of the “A-area” and the “B-area” the position of a pointer ispresent in, the UI object 30 may be moved. For example, when a pointeris displayed at a predetermined position (e.g., the center) in astereoscopic image, and the pointer is present in the A-area, the above“movement process in the A-area” may be performed. When the pointer ispresent in the B-area, the above “movement process in the B-area” may beperformed.

In the above exemplary embodiment, when a condition is satisfied, the UIobject 30 is moved. An object to be moved may be not only the UI object30 but also any object. For example, when a condition is satisfied, acharacter object conveying predetermined information to the user may bemoved to the front of the virtual camera VC.

In the above exemplary embodiment, the orientation of the goggleapparatus 10 (the information processing apparatus 2) is detected basedon data from the inertial sensor 22 included in the informationprocessing apparatus 2. In another exemplary embodiment, the orientationof the goggle apparatus 10 may be detected by another method. Forexample, the image display system 1 may include a camera that externallycaptures the goggle apparatus 10, capture the goggle apparatus 10 or amarker attached to the goggle apparatus 10 using the camera, and acquirethe orientation of the goggle apparatus 10 based on the captured image.The goggle apparatus 10 may include a camera, and the orientation of thegoggle apparatus 10 may be acquired based on a change in an imagecaptured by the camera.

In the above exemplary embodiment, the rotation direction of the line ofsight of the virtual camera VC is divided into a component in theleft-right direction (the yaw direction) and a component in the up-downdirection (the pitch direction) in the virtual space, and based on achange in the line of sight of the virtual camera VC from one directionto the other direction regarding the component in the left-rightdirection or the up-down direction, the UI object 30 is moved. Forexample, when the line of sight of the virtual camera VC changes fromone direction (e.g., the left direction) to the other direction (e.g.,the right direction) regarding the component in the left-rightdirection, the amounts of rotation in the other direction and/or the onedirection regarding the component in the left-right direction may becalculated. When the calculated amounts of rotation are greater than orequal to a predetermined value (which may or may not include thepredetermined value), the UI object 30 is moved. In another exemplaryembodiment, based on a change in the line of sight of the virtual cameraVC from one direction (e.g., the direction from the lower left to theupper right) to the other direction (e.g., the direction from the upperright to the lower left) in an oblique direction, the UI object 30 maybe moved.

That is, based on a change in a predetermined component (e.g., acomponent in the left-right direction, a component in the up-downdirection, or a component in an oblique direction) of the rotationdirection of the line of sight of the virtual camera VC from onedirection to the other direction, the UI object 30 may be moved.

In the above exemplary embodiment, when the line of sight of the virtualcamera VC is present in the A-area, the UI object 30 is moved followingthe rotation of the virtual camera VC in the left-right direction. Inanother exemplary embodiment, after the line of sight of the virtualcamera VC comes out of the UI area and enters the A-area, and at thetiming when the line of sight of the virtual camera VC returns to the UIarea, the UI object 30 may be moved to the front of the virtual cameraVC in the left-right direction (the yaw direction) (or the virtualcamera VC may be controlled). That is, the UI object 30 may be moved atany timing so long as after the user removes the goggle apparatus 10(e.g., after the user directs the bottom surface 15 of the goggleapparatus 10 in the up or down direction), and when the user views thedisplay section by putting the goggle apparatus 10 on their face, the UIobject 30 is controlled to be seen in front. For example, at the timewhen the line of sight of the virtual camera VC enters the A-area, theUI object 30 may be moved following the rotation of the virtual cameraVC in the left-right direction. Alternatively, at the time when the lineof sight of the virtual camera VC enters the A-area and returns to theUI area, the UI object 30 may be moved to the front of the virtualcamera VC in the left-right direction.

In the above exemplary embodiment, when the line of sight of the virtualcamera VC is present in the A-area, the UI object 30 is controlled to belocated in front of the virtual camera VC in the yaw direction. In acase where the line of sight of the virtual camera VC is present in theB-area, under the condition that the line of sight of the virtual cameraVC reverses, the UI object 30 is controlled to be located in front ofthe virtual camera VC. “Front” as used herein includes not only theexact front but also the substantial front. The exact front means that astraight line extending from the line of sight of the virtual camera VCintersects a straight line parallel to the y-axis passing through thecenter in the left-right direction of the UI object 30. In other words,the “exact front” means that the line of sight of the virtual camera VCis not shifted in the left-right direction of the UI object 30. The“substantial front” also includes a case where the line of sight of thevirtual camera VC is shifted in the left-right direction of the UIobject 30. For example, “the UI object 30 is located at the substantialfront of the virtual camera VC” may mean that a surface perpendicular tothe xz plane passing through the line of sight of the virtual camera VCintersects the UI object 30. In this case, when the rotational angle ofthe virtual camera VC in the pitch direction relative to the UI object30 is “0” degrees, a straight line extending from the line of sight ofthe virtual camera VC hits the UI object 30. “The UI object is moved sothat the UI object 30 is located at the substantial front of the virtualcamera VC” may mean that when the rotational angle of the virtual cameraVC in the pitch direction relative to the UI object 30 is “0” degrees,the UI object is moved so that at least a part of the UI object 30enters the field of view of the virtual camera VC, and the rotationalangle of the virtual camera VC in the yaw direction relative to the UIobject 30 is smaller than before the movement of the UI object. That is,“front” as used herein means that the rotational angle of the virtualcamera in the yaw direction relative to the UI object (a predeterminedobject) is a predetermined angle (a certain angle). The “predeterminedangle” may be such an angle that when the rotational angle of thevirtual camera in the pitch direction relative to the predeterminedobject is “0” degrees, at least a part of the predetermined object isincluded in the field of view of the virtual camera.

In the above exemplary embodiment, when the line of sight of the virtualcamera VC is present in the A-area above the UI area or the A-area belowthe UI area, the above movement process in the A-area is performed. Inanother exemplary embodiment, only when the line of sight of the virtualcamera VC is present in the A-area above the UI area, the above movementprocess in the A-area may be performed. Conversely, only when the lineof sight of the virtual camera VC is present in the A-area below the UIarea, the above movement process in the A-area may be performed.

The size and the shape of each area shown in FIG. 9 are merely examples,and each area may have any other size and shape. When the line of sightof the virtual camera VC is present at the boundary between areas, itmay be determined that the line of sight of the virtual camera VC ispresent in the UI area, or it may be determined that the line of sightof the virtual camera VC is present in an area (i.e., the A-area or theB-area) different from the UI area.

In the above exemplary embodiment, based on two virtual cameras, namelythe left virtual camera VCL and the right virtual camera VCR, a left eyeimage and a right eye image having parallax with each other aregenerated. In another exemplary embodiment, an image may be generatedbased on a single virtual camera, and a left eye image and a right eyeimage having parallax with each other may be generated by deforming thegenerated image. That is, “generating a left eye image and a right eyeimage based on a virtual camera” in the present specification includesboth generating a left eye image and a right eye image based on a pairof the left virtual camera VCL and the right virtual camera VCR (i.e.,based on a plurality of virtual cameras), and generating a left eyeimage and a right eye image based on a single virtual camera.

In the above exemplary embodiment, the display section 21 of the goggleapparatus 10 is attachable to and detachable from the goggle main body.In another exemplary embodiment, the display section 21 of the goggleapparatus 10 may be fixed to the goggle main body. That is, the goggleapparatus 10 may be obtained by integrally forming a display section fordisplaying a left-eye image and a right-eye image and an informationprocessing apparatus for generating the left-eye image and the right-eyeimage. The display section of the goggle apparatus 10 is not limited tothe display section 21 as described above. For example, the displaysection of the goggle apparatus 10 may include two display sections (aleft eye display section viewed by the left eye of the user and a righteye display section viewed by the right eye of the user). The displaysection of the goggle apparatus 10 may have any shape. For example, thedisplay section itself of the goggle apparatus 10 may be formed into anapproximate circle (a circle or an ellipse) as in the left opening 16Land the right opening 16R. Two left and right display sections formedinto squares or rectangles may be used as the display section of thegoggle apparatus 10. The display section of the goggle apparatus 10 maybe a display device such as a liquid crystal display device or anorganic EL display device, or may be a display device using a projectionmethod for projecting a video onto a projection surface.

The configuration of the image display system 1 according to the aboveexemplary embodiment is merely an example, and is not limited to theabove. For example, in the above exemplary embodiment, the goggleapparatus 10 is formed of the information processing apparatus 2including the display section 21 that displays an image and theprocessor 20 that performs the process of generating an image, and thegoggle main body. That is, the image display system 1 is formed of thegoggle apparatus 10 including the goggle main body and the informationprocessing apparatus 2. In another exemplary embodiment, a goggleapparatus including a display section (a goggle apparatus integratedwith a display section) and an information processing apparatus thatperforms the process of generating an image may be formed as separateapparatuses, and the image display system 1 may be formed of theseplurality of apparatuses. In this case, the goggle apparatus and theinformation processing apparatus may be connected together in a wired orwireless manner, and a left eye image and a right eye image generated bythe information processing apparatus may be transmitted to the goggleapparatus and viewed by the user. The information processing apparatusmay perform a process regarding the movement of the UI object 30described above and transmit the result of the process to the goggleapparatus, the goggle apparatus may generate a left eye image and aright eye image, and the left eye image and the right eye image may beviewed by the user. The goggle apparatus and the information processingapparatus may be connected together via a network (a LAN, a WAN, theInternet, or the like).

In the exemplary embodiment, the goggle apparatus is used in which theuser looks into the display section while keeping holding the goggleapparatus with their hand. In another exemplary embodiment, the goggleapparatus may not need to be held by the hand of the user, and may be ahead-mounted display fixedly attached to the head of the user.

While the exemplary embodiment has been described, the above descriptionis merely illustrative, and the exemplary embodiment may be improved andmodified in various manners.

While certain example systems, methods, devices and apparatuses havebeen described herein, it is to be understood that the appended claimsare not to be limited to the systems, methods, devices and apparatusesdisclosed, but on the contrary, are intended to cover variousmodifications and equivalent arrangements included within the spirit andscope of the appended claims.

1-13. (canceled)
 14. An image display system, comprising: a goggleapparatus; and processing circuitry including at least one processor,the processing circuitry configured to at least: place an object in avirtual space; display, on a display section of the goggle apparatus, animage captured by a virtual camera in the virtual space; acquire atleast a rotational angle in a yaw direction and a rotational angle in apitch direction of the goggle apparatus; rotate the virtual camera in ayaw direction in the virtual space in accordance with the rotationalangle in the yaw direction of the goggle apparatus and rotate thevirtual camera in a pitch direction in the virtual space in accordancewith the rotational angle in the pitch direction of the goggleapparatus; and if at least the rotational angle in the pitch directionof the goggle apparatus is outside a range and/or the rotational anglein the yaw direction of the goggle apparatus satisfies a predeterminedcondition, perform at least either one of a movement of the object andcontrol of the virtual camera in the virtual space so that the object islocated in front of the virtual camera in the yaw direction.
 15. Theimage display system according to claim 14, wherein while the rotationalangle in the pitch direction of the goggle apparatus is outside therange, at least one of the movement of the object and/or the control ofthe virtual camera is performed in accordance with rotation of thegoggle apparatus in the yaw direction so that the object continues to belocated in front of the virtual camera in the yaw direction.
 16. Theimage display system according to claim 14, wherein the object isselected based on, at least, the rotational angle in the pitch directionof the goggle apparatus, and if the rotational angle in the pitchdirection of the goggle apparatus is outside the range, the object isnot selected.
 17. The image display system according to claim 14,wherein if an angle of depression of the goggle apparatus is greaterthan or equal to a threshold, or if an angle of elevation of the goggleapparatus is greater than or equal to the threshold, the rotationalangle in the pitch direction of the goggle apparatus is determined asbeing outside the range.
 18. The image display system according to claim14, wherein if the rotational angle in the pitch direction of the goggleapparatus is outside the range, the object is moved to the front of thevirtual camera in the yaw direction while maintaining a position of theobject in a height direction in the virtual space.
 19. The image displaysystem according to claim 14, wherein the object includes a userinterface that can be operated by a user.
 20. The image display systemaccording to claim 14, wherein in a case where the rotational angle inthe pitch direction of the goggle apparatus is outside the range, andwhen the rotational angle in the pitch direction of the goggle apparatuschanges from outside the range to within the range, at least any of themovement of the object and/or the control of the virtual camera isperformed so that the object is in front of the virtual camera in theyaw direction.
 21. The image display system according to claim 14,wherein if at least the rotational angle in the pitch direction of thegoggle apparatus is outside the range indicating movement from a firstdirection reversing to a second direction, at least any of the movementof the object and/or control of the virtual camera in the virtual spaceis performed so that the object is located in front of the virtualcamera in the yaw direction.
 22. The image display system according toclaim 21, wherein when a sum of an amount of change from the firstdirection to the second direction is greater than or equal to apredetermined threshold, movement of the object and/or control of thevirtual camera in the virtual space is performed so that the object islocated in front of the virtual camera in the yaw direction.
 23. Theimage display system according to claim 21, wherein when a sum of anamount of change from the first direction to the second direction isless than a predetermined threshold, movement of the object and/orcontrol of the virtual camera in the virtual space is not performed. 24.The image display system according to claim 14, wherein the object is auser interface image positioned in a user interface area of the virtualspace, and when the rotational angle in the pitch direction of thegoggle apparatus is outside the range and/or the rotational angle in theyaw direction of the goggle apparatus satisfies a predeterminedcondition, a line of sight of the virtual camera is outside of the userinterface area of the virtual space.
 25. The image display systemaccording to claim 14, wherein the object is a user interface imagepositioned in a user interface area of the virtual space, and when therotational angle in the pitch direction of the goggle apparatus iswithin the range, a line of sight of the virtual camera is presentwithin the user interface area of the virtual space.
 26. Anon-transitory storage medium having stored therein an image displayprogram executed by a processor of an apparatus that displays an imageon a display section of a goggle apparatus, the image display programcausing the processor to provide execution comprising: placing an objectin a virtual space; displaying, on the display section of the goggleapparatus, an image captured by a virtual camera in the virtual space;acquiring at least a rotational angle in a yaw direction and arotational angle in a pitch direction of the goggle apparatus; rotatingthe virtual camera in a yaw direction in the virtual space in accordancewith the rotational angle in the yaw direction of the goggle apparatusand rotating the virtual camera in a pitch direction in the virtualspace in accordance with the rotational angle in the pitch direction ofthe goggle apparatus; and if at least the rotational angle in the pitchdirection of the goggle apparatus is outside a range and/or therotational angle in the yaw direction of the goggle apparatus satisfiesa predetermined condition, performing at least either one of a movementof the object and control of the virtual camera in the virtual space sothat the object is located in front of the virtual camera in the yawdirection.
 27. A display control apparatus configured to display animage on a display section of a goggle apparatus, the display controlapparatus comprising: a processor; and a memory configured to storecomputer readable instructions that, when executed by the processor,cause the display control apparatus to: place an object in a virtualspace; display, on the display section of the goggle apparatus, an imagecaptured by a virtual camera in the virtual space; acquire at least arotational angle in a yaw direction and a rotational angle in a pitchdirection of the goggle apparatus; rotate the virtual camera in a yawdirection in the virtual space in accordance with the rotational anglein the yaw direction of the goggle apparatus and rotate the virtualcamera in a pitch direction in the virtual space in accordance with therotational angle in the pitch direction of the goggle apparatus; and ifat least the rotational angle in the pitch direction of the goggleapparatus is outside a range and/or the rotational angle in the yawdirection of the goggle apparatus satisfies a predetermined condition,perform at least either one of a movement of the object and control ofthe virtual camera in the virtual space so that the object is located infront of the virtual camera in the yaw direction.
 28. An image displaymethod performed by an image display system including a goggleapparatus, the image display method comprising: placing an object in avirtual space; displaying, on a display section of the goggle apparatus,an image captured by a virtual camera in the virtual space; acquiring atleast a rotational angle in a yaw direction and a rotational angle in apitch direction of the goggle apparatus; rotating the virtual camera ina yaw direction in the virtual space in accordance with the rotationalangle in the yaw direction of the goggle apparatus and rotating thevirtual camera in a pitch direction in the virtual space in accordancewith the rotational angle in the pitch direction of the goggleapparatus; and if at least the rotational angle in the pitch directionof the goggle apparatus is outside a range and/or the rotational anglein the yaw direction of the goggle apparatus satisfies a predeterminedcondition, performing at least either one of a movement of the objectand control of the virtual camera in the virtual space so that theobject is located in front of the virtual camera in the yaw direction.